A cache memory is a relatively small fast memory which may be viewed as a buffer memory for a main or system memory and the access time to a cache memory is many times less than the access time to a main memory. The performance goal of adding a cache memory to a computer is to make the average memory access time as seen by the processor as close as possible to that of the cache memory.
The main advantage of a cache memory is that it reduces the bus traffic between a CPU (Central Processing Unit) and its system memory. In general, a cache memory contains a copy of some of the information in system memory. Although the algorithms used to read and fill memory caches vary, the common feature is that they take advantage of the temporal and/or spacial locality of the code or data accessed by the CPU. The predictability of logical memory addresses which is essential to the successful operation of a cache-main memory hierarchy is based on a common characteristic of computer programs called locality of reference. This describes the fact that over the short term, the addresses generated by a typical program tend to be confined to small regions of its logical address space.
The performance of a two-level (cache-main) memory hierarchy is frequently measured in terms of the hit ratio which is defined as the probability that a logical address generated by a CPU refers to information available from the cache memory. If the address generated by a CPU refers to information only available from main memory, that event is called a cache miss.
The contemporary VLSI processors may be divided into two sets; those that have a simple memory interface without special cache control, and those that support their own cache interface. The former group may have an external cache added to the chip boundary which can then supply the requested word in a much shorter time, if the requested address matches one that is stored in the cache. In this situation the processor does not distinguish between cache and non-cache cycles; the only difference between the two is that the cached cycle response time is much shorter.
Some processors however, support their own cache interface, and thus do distinguish between cached and non-cached accesses e.g. the R2000 CPU available from MIPS Computer Systems Inc., Sunnyvale, Calif. The CPU determines whether or not the requested address is resident in the cache (hit condition), and if not it initiates a cache miss cycle to request the information from system memory.
Common to both situations is that the address is grouped into two halves; the lower half addresses into the cache array, and the upper half is returned from the cache as data referred to as the tag. If the tag matches the requested upper address half, then the request is a hit, and the data from the cache is valid. If the tag does not match the requested upper address half, then the CPU enters a miss cycle that causes the main memory to be accessed and the retrieved information to be written in the cache memory at the originally addressed location. The simplest cache scheme allocates one tag for each word address. This type of cache is known as a direct-mapped cache memory.
Caches are very often organized into multiple sets, where all the sets are addressed simultaneously by the CPU. This has been demonstrated to be more efficient for a given amount of cache memory. In many applications the effectiveness of caches usually drops exponentially after a few Kbytes, meaning that extra memory is then better applied to allocating additional sets to increase the likelihood of hits occurring while minimizing the possibility that a needed address will be overwritten by another cached entry. This cache memory organization is referred to as a set-associative cache memory. However, since all entries must be checked simultaneously for a tag hit, this scheme is more complex than a direct-mapped arrangement. In some cases, the additional control circuitry that must be added to provide the set-associative configuration may tend to compromise the cache memory access time.